{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/NeuromatchAcademy/course-content/blob/master/tutorials/W0D3_LinearAlgebra/student/W0D3_Tutorial2.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Neuromatch Academy: Week 0, Day 3, Tutorial 2\n",
    "# Linear Algebra: Matrices\n",
    "\n",
    "__Content creators:__ Ella Batty\n",
    "\n",
    "\n",
    "__Content reviewers:__ Name Surname, Name Surname. \n",
    "\n",
    "__Content editors:__ Name Surname, Name Surname.\n",
    "\n",
    "__Production editors:__ Siddharth Suresh  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "#Tutorial Objectives\n",
    "During today, we will learn the basics of linear algebra, focusing on the topics that underlie the material on future days in the NMA Computational Neuroscience course. In this tutorial, we focus on matrices: their definition, their properties & operations, and especially on a geometrical intuition of them. \n",
    "\n",
    "By the end of this tutorial, you will be able to :\n",
    "* Explain matrices as a linear transformation and relate matrix properties to properties of that linear transformation\n",
    "*  Perform matrix multiplication by hand\n",
    "*  Define what eigenvalues/eigenvectors are\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Code Credit:**\n",
    "\n",
    "Some elements of this problem set are from or inspired by https://openedx.seas.gwu.edu/courses/course-v1:GW+EngComp4+2019/about. In particular, we are using their `plot_linear_transformation` and `plot_linear_transformations` functions.\n",
    "\n",
    " Code under BSD 3-Clause License © 2019 Lorena A. Barba, Tingyu Wang. THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:22.173516Z",
     "iopub.status.busy": "2021-05-30T19:06:22.172969Z",
     "iopub.status.idle": "2021-05-30T19:06:22.461511Z",
     "shell.execute_reply": "2021-05-30T19:06:22.460437Z"
    }
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:22.467338Z",
     "iopub.status.busy": "2021-05-30T19:06:22.466238Z",
     "iopub.status.idle": "2021-05-30T19:06:22.545605Z",
     "shell.execute_reply": "2021-05-30T19:06:22.544718Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Figure settings\n",
    "import ipywidgets as widgets# interactive display\n",
    "from ipywidgets import fixed\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "plt.style.use(\"https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:22.579989Z",
     "iopub.status.busy": "2021-05-30T19:06:22.558849Z",
     "iopub.status.idle": "2021-05-30T19:06:22.582371Z",
     "shell.execute_reply": "2021-05-30T19:06:22.581724Z"
    }
   },
   "outputs": [],
   "source": [
    "# @title Plotting functions\n",
    "import numpy\n",
    "from numpy.linalg import inv, eig\n",
    "from math import ceil\n",
    "from matplotlib import pyplot, ticker, get_backend, rc\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "from itertools import cycle\n",
    "\n",
    "_int_backends = ['GTK3Agg', 'GTK3Cairo', 'MacOSX', 'nbAgg',\n",
    "                 'Qt4Agg', 'Qt4Cairo', 'Qt5Agg', 'Qt5Cairo',\n",
    "                 'TkAgg', 'TkCairo', 'WebAgg', 'WX', 'WXAgg', 'WXCairo']\n",
    "_backend = get_backend()   # get current backend name\n",
    "\n",
    "# shrink figsize and fontsize when using %matplotlib notebook\n",
    "if _backend in _int_backends:\n",
    "  fontsize = 4\n",
    "  fig_scale = 0.75\n",
    "else:\n",
    "  fontsize = 5\n",
    "  fig_scale = 1\n",
    "\n",
    "grey = '#808080'\n",
    "gold = '#cab18c'   # x-axis grid\n",
    "lightblue = '#0096d6'  # y-axis grid\n",
    "green = '#008367'  # x-axis basis vector\n",
    "red = '#E31937'    # y-axis basis vector\n",
    "darkblue = '#004065'\n",
    "\n",
    "pink, yellow, orange, purple, brown = '#ef7b9d', '#fbd349', '#ffa500', '#a35cff', '#731d1d'\n",
    "\n",
    "quiver_params = {'angles': 'xy',\n",
    "                 'scale_units': 'xy',\n",
    "                 'scale': 1,\n",
    "                 'width': 0.012}\n",
    "\n",
    "grid_params = {'linewidth': 0.5,\n",
    "               'alpha': 0.8}\n",
    "\n",
    "def set_rc(func):\n",
    "  def wrapper(*args, **kwargs):\n",
    "    rc('font', family='serif', size=fontsize)\n",
    "    rc('figure', dpi=200)\n",
    "    rc('axes', axisbelow=True, titlesize=5)\n",
    "    rc('lines', linewidth=1)\n",
    "    func(*args, **kwargs)\n",
    "  return wrapper\n",
    "\n",
    "@set_rc\n",
    "def plot_vector(vectors, tails=None):\n",
    "  ''' Draw 2d vectors based on the values of the vectors and the position of their tails.\n",
    "\n",
    "  Parameters\n",
    "  ----------\n",
    "  vectors : list.\n",
    "    List of 2-element array-like structures, each represents a 2d vector.\n",
    "\n",
    "  tails : list, optional.\n",
    "    List of 2-element array-like structures, each represents the coordinates of the tail\n",
    "    of the corresponding vector in vectors. If None (default), all tails are set at the\n",
    "    origin (0,0). If len(tails) is 1, all tails are set at the same position. Otherwise,\n",
    "    vectors and tails must have the same length.\n",
    "\n",
    "  Examples\n",
    "  --------\n",
    "  >>> v = [(1, 3), (3, 3), (4, 6)]\n",
    "  >>> plot_vector(v)      # draw 3 vectors with their tails at origin\n",
    "  >>> t = [numpy.array((2, 2))]\n",
    "  >>> plot_vector(v, t)   # draw 3 vectors with their tails at (2,2)\n",
    "  >>> t = [[3, 2], [-1, -2], [3, 5]]\n",
    "  >>> plot_vector(v, t)   # draw 3 vectors with 3 different tails\n",
    "\n",
    "  '''\n",
    "  vectors = numpy.array(vectors)\n",
    "  assert vectors.shape[1] == 2, \"Each vector should have 2 elements.\"\n",
    "  if tails is not None:\n",
    "    tails = numpy.array(tails)\n",
    "    assert tails.shape[1] == 2, \"Each tail should have 2 elements.\"\n",
    "  else:\n",
    "    tails = numpy.zeros_like(vectors)\n",
    "\n",
    "  # tile vectors or tails array if needed\n",
    "  nvectors = vectors.shape[0]\n",
    "  ntails = tails.shape[0]\n",
    "  if nvectors == 1 and ntails > 1:\n",
    "    vectors = numpy.tile(vectors, (ntails, 1))\n",
    "  elif ntails == 1 and nvectors > 1:\n",
    "    tails = numpy.tile(tails, (nvectors, 1))\n",
    "  else:\n",
    "    assert tails.shape == vectors.shape, \"vectors and tail must have a same shape\"\n",
    "\n",
    "  # calculate xlimit & ylimit\n",
    "  heads = tails + vectors\n",
    "  limit = numpy.max(numpy.abs(numpy.hstack((tails, heads))))\n",
    "  limit = numpy.ceil(limit * 1.2)   # add some margins\n",
    "\n",
    "  figsize = numpy.array([2,2]) * fig_scale\n",
    "  figure, axis = pyplot.subplots(figsize=figsize)\n",
    "  axis.quiver(tails[:,0], tails[:,1], vectors[:,0], vectors[:,1], color=darkblue,\n",
    "                  angles='xy', scale_units='xy', scale=1)\n",
    "  axis.set_xlim([-limit, limit])\n",
    "  axis.set_ylim([-limit, limit])\n",
    "  axis.set_aspect('equal')\n",
    "\n",
    "  # if xticks and yticks of grid do not match, choose the finer one\n",
    "  xticks = axis.get_xticks()\n",
    "  yticks = axis.get_yticks()\n",
    "  dx = xticks[1] - xticks[0]\n",
    "  dy = yticks[1] - yticks[0]\n",
    "  base = max(int(min(dx, dy)), 1)   # grid interval is always an integer\n",
    "  loc = ticker.MultipleLocator(base=base)\n",
    "  axis.xaxis.set_major_locator(loc)\n",
    "  axis.yaxis.set_major_locator(loc)\n",
    "  axis.grid(True, **grid_params)\n",
    "\n",
    "  # show x-y axis in the center, hide frames\n",
    "  axis.spines['left'].set_position('center')\n",
    "  axis.spines['bottom'].set_position('center')\n",
    "  axis.spines['right'].set_color('none')\n",
    "  axis.spines['top'].set_color('none')\n",
    "\n",
    "@set_rc\n",
    "def plot_transformation_helper(axis, matrix, *vectors, unit_vector=True, unit_circle=False, title=None):\n",
    "  \"\"\" A helper function to plot the linear transformation defined by a 2x2 matrix.\n",
    "\n",
    "  Parameters\n",
    "  ----------\n",
    "  axis : class matplotlib.axes.Axes.\n",
    "    The axes to plot on.\n",
    "\n",
    "  matrix : class numpy.ndarray.\n",
    "    The 2x2 matrix to visualize.\n",
    "\n",
    "  *vectors : class numpy.ndarray.\n",
    "    The vector(s) to plot along with the linear transformation. Each array denotes a vector's\n",
    "    coordinates before the transformation and must have a shape of (2,). Accept any number of vectors.\n",
    "\n",
    "  unit_vector : bool, optional.\n",
    "    Whether to plot unit vectors of the standard basis, default to True.\n",
    "\n",
    "  unit_circle: bool, optional.\n",
    "    Whether to plot unit circle, default to False.\n",
    "\n",
    "  title: str, optional.\n",
    "    Title of the plot.\n",
    "\n",
    "  \"\"\"\n",
    "  assert matrix.shape == (2,2), \"the input matrix must have a shape of (2,2)\"\n",
    "  grid_range = 20\n",
    "  x = numpy.arange(-grid_range, grid_range+1)\n",
    "  X_, Y_ = numpy.meshgrid(x,x)\n",
    "  I = matrix[:,0]\n",
    "  J = matrix[:,1]\n",
    "  X = I[0]*X_ + J[0]*Y_\n",
    "  Y = I[1]*X_ + J[1]*Y_\n",
    "  origin = numpy.zeros(1)\n",
    "\n",
    "  # draw grid lines\n",
    "  for i in range(x.size):\n",
    "    axis.plot(X[i,:], Y[i,:], c=gold, **grid_params)\n",
    "    axis.plot(X[:,i], Y[:,i], c=lightblue, **grid_params)\n",
    "\n",
    "  # draw (transformed) unit vectors\n",
    "  if unit_vector:\n",
    "    axis.quiver(origin, origin, [I[0]], [I[1]], color=green, **quiver_params)\n",
    "    axis.quiver(origin, origin, [J[0]], [J[1]], color=red, **quiver_params)\n",
    "\n",
    "  # draw optional vectors\n",
    "  color_cycle = cycle([pink, darkblue, orange, purple, brown])\n",
    "  if vectors:\n",
    "    for vector in vectors:\n",
    "      color = next(color_cycle)\n",
    "      vector_ = matrix @ vector.reshape(-1,1)\n",
    "      axis.quiver(origin, origin, [vector_[0]], [vector_[1]], color=color, **quiver_params)\n",
    "\n",
    "  # draw optional unit circle\n",
    "  if unit_circle:\n",
    "    alpha =  numpy.linspace(0, 2*numpy.pi, 41)\n",
    "    circle = numpy.vstack((numpy.cos(alpha), numpy.sin(alpha)))\n",
    "    circle_trans = matrix @ circle\n",
    "    axis.plot(circle_trans[0], circle_trans[1], color=red, lw=0.8)\n",
    "\n",
    "  # hide frames, set xlimit & ylimit, set title\n",
    "  limit = 4\n",
    "  axis.spines['left'].set_position('center')\n",
    "  axis.spines['bottom'].set_position('center')\n",
    "  axis.spines['left'].set_linewidth(0.3)\n",
    "  axis.spines['bottom'].set_linewidth(0.3)\n",
    "  axis.spines['right'].set_color('none')\n",
    "  axis.spines['top'].set_color('none')\n",
    "  axis.set_xlim([-limit, limit])\n",
    "  axis.set_ylim([-limit, limit])\n",
    "  if title is not None:\n",
    "    axis.set_title(title)\n",
    "\n",
    "@set_rc\n",
    "def plot_linear_transformation(matrix, *vectors, name = None, unit_vector=True, unit_circle=False):\n",
    "  \"\"\" Plot the linear transformation defined by a 2x2 matrix using the helper\n",
    "  function plot_transformation_helper(). It will create 2 subplots to visualize some\n",
    "  vectors before and after the transformation.\n",
    "\n",
    "  Parameters\n",
    "  ----------\n",
    "  matrix : class numpy.ndarray.\n",
    "    The 2x2 matrix to visualize.\n",
    "\n",
    "  *vectors : class numpy.ndarray.\n",
    "    The vector(s) to plot along with the linear transformation. Each array denotes a vector's\n",
    "    coordinates before the transformation and must have a shape of (2,). Accept any number of vectors.\n",
    "\n",
    "  unit_vector : bool, optional.\n",
    "    Whether to plot unit vectors of the standard basis, default to True.\n",
    "\n",
    "  unit_circle: bool, optional.\n",
    "    Whether to plot unit circle, default to False.\n",
    "\n",
    "  \"\"\"\n",
    "  figsize = numpy.array([4,2]) * fig_scale\n",
    "  figure, (axis1, axis2) = pyplot.subplots(1, 2, figsize=figsize)\n",
    "  plot_transformation_helper(axis1, numpy.identity(2), *vectors, unit_vector=unit_vector, unit_circle=unit_circle, title='Before transformation')\n",
    "  plot_transformation_helper(axis2, matrix, *vectors, unit_vector=unit_vector, unit_circle=unit_circle, title='After transformation')\n",
    "  if name is not None:\n",
    "    figure.suptitle(f'Population {name}')\n",
    "\n",
    "\n",
    "\n",
    "def plot_eig_vec_transform(W):\n",
    "  classic = 'k'\n",
    "  vec_names = ['a', 'b','c','d','e','f','g', 'h']\n",
    "\n",
    "  _, vecs = np.linalg.eig(W)\n",
    "  vecs = vecs.T\n",
    "\n",
    "  fig, axes = plt.subplots(1, 2, figsize=(2, 1))\n",
    "  colors = plt.rcParams['axes.prop_cycle'].by_key()['color']\n",
    "\n",
    "  for i in range(2):\n",
    "    axes[i].set(xlim=[-3.5, 3.5], ylim=[-3.5,3.5])\n",
    "    axes[i].axis('Off')\n",
    "    axes[i].plot([0, 0], [-3.5, 3.5], classic, alpha=.4)\n",
    "    axes[i].plot([-3.5, 3.5], [0, 0], classic, alpha=.4)\n",
    "\n",
    "  for i_vec, vec in enumerate(vecs):\n",
    "    axes[0].arrow(0, 0, vec[0], vec[1], head_width=.2, facecolor=colors[i_vec], edgecolor=colors[i_vec], length_includes_head=True)\n",
    "    axes[0].annotate(vec_names[i_vec], xy=(vec[0]+np.sign(vec[0])*.15, vec[1]+np.sign(vec[1])*.15), color=colors[i_vec])\n",
    "\n",
    "    transformed_vec = np.matmul(W, vec)\n",
    "    axes[1].arrow(0, 0, transformed_vec[0], transformed_vec[1], head_width=.2, facecolor=colors[i_vec], edgecolor=colors[i_vec], length_includes_head=True)\n",
    "    axes[1].annotate(vec_names[i_vec], xy=(transformed_vec[0]+np.sign(transformed_vec[0])*.15, transformed_vec[1]+np.sign(transformed_vec[1])*.15), color=colors[i_vec])\n",
    "\n",
    "  axes[0].set_title('Before')\n",
    "  axes[1].set_title('After')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "# Section 1: Intro to matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1.1: Matrices to solve systems of equations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:22.589891Z",
     "iopub.status.busy": "2021-05-30T19:06:22.589369Z",
     "iopub.status.idle": "2021-05-30T19:06:22.619432Z",
     "shell.execute_reply": "2021-05-30T19:06:22.619777Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 1: Systems of Equations\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"Py04T8qSl-4\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In a variety of contexts, we may encounter systems of linear equations like this one:\n",
    "\n",
    "$$ \n",
    "\\begin{align}\n",
    "3x_1 + 2x_2 + x_3 &= y_1 \\\\\n",
    "7x_1 + x_2 + 2x_3 &= y_2 \\\\\n",
    "x_1 - x_2  - 2x_3 &= y_3 \\\\\n",
    "\\end{align}$$\n",
    "We may know all the x's and want to solve for y's, or we may know the y's and want to solve for the x's. We can solve this in several different ways but one especially appealing way is to cast it as a matrix-vector equation:\n",
    "\n",
    "$$\\mathbf{W}\\mathbf{x} = \\mathbf{y}$$\n",
    "where \n",
    "$$\\begin{align}\n",
    "\\mathbf{W} &= \\begin{bmatrix} 3 & 2 & 1 \\\\ 7 & 1 & 2 \\\\ 1 &-1 &-2 \\end{bmatrix}, \\mathbf{x} = \\begin{bmatrix} x_1 \\\\ x_2 \\\\ x_3 \\end{bmatrix}, \\mathbf{y} = \\begin{bmatrix} y_1 \\\\ y_2 \\\\ y_3 \\end{bmatrix}\\\\\n",
    "\\end{align}$$\n",
    "\n",
    "If we know $\\mathbf{W}$ and $\\mathbf{x}$, we can solve for $\\mathbf{y}$ using matrix-vector multiplication. Each row of $\\mathbf{y}$ is computed as the dot product of the equivalent row of $\\mathbf{W}$ and $\\mathbf{x}$. \n",
    "\n",
    "If we know $\\mathbf{W}$ and $\\mathbf{y}$, we can sometimes solve for $\\mathbf{x}$ by using the inverse of $\\mathbf{W}$:\n",
    "$$ \\mathbf{x} = W^{-1}\\mathbf{y} $$.\n",
    "The reason this only sometimes works will be dived into later in this tutorial!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Coding Exercise 1.1: Understanding neural transformations\n",
    "\n",
    "We will look at a group of 2 V1 neurons which get input from 2 LGN neurons: we will call these 2 V1 neurons population p. Below, we have the system of linear equations that dictates the neuron models for each population. $g_1$ and $g_2$ correspond to the LGN neural activities (of neuron 1 and 2). $V_{p_1}$ and  $V_{p_2}$ correspond to the responses of the V1 neurons 1 and 2 in population p. \n",
    "\n",
    "$$\\begin{align}\n",
    "g_1 + 3g_2 &= V_{p_1} \\\\\n",
    "2g_1 + g_2 &= V_{p_2} \\\\\n",
    "\\end{align}$$\n",
    "\n",
    "\n",
    "\n",
    "1) Cast each this as a matrix-vector multiplication: \n",
    "\n",
    "$$ \\mathbf{v}_p = \\mathbf{P}\\mathbf{g} $$\n",
    "where P is the weight matrix to population p. \n",
    "\n",
    "2) Let's say we only recorded from the V1 cells (and know the weight matrix) and are trying to figure out how the LGN cells responded. Solve the matrix equation for the given V1 activities:\n",
    "\n",
    "$$\\mathbf{v}_p = \\begin{bmatrix}\n",
    "16 \\\\\n",
    "7 \\\\\n",
    "\\end{bmatrix}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:22.624305Z",
     "iopub.status.busy": "2021-05-30T19:06:22.623188Z",
     "iopub.status.idle": "2021-05-30T19:06:22.625881Z",
     "shell.execute_reply": "2021-05-30T19:06:22.625460Z"
    }
   },
   "outputs": [],
   "source": [
    "# Create P (using np array)\n",
    "P = ...\n",
    "\n",
    "# Create v_p (using np array)\n",
    "v_p = ...\n",
    "\n",
    "# Solve for g (using np.linalg.inv)\n",
    "g = ...\n",
    "\n",
    "# Print g\n",
    "print(g)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:22.634288Z",
     "iopub.status.busy": "2021-05-30T19:06:22.633402Z",
     "iopub.status.idle": "2021-05-30T19:06:22.635398Z",
     "shell.execute_reply": "2021-05-30T19:06:22.633799Z"
    }
   },
   "source": [
    "[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_2147a3e1.py)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see the output [1, 5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can recover how the LGN neurons respond given the weight matrix and V1 responses! You have solved the system of equations using matrices. We can't always do this though: let's say we have a different group of 2 V1 neurons -  population q - with the following weight matrix from the LGN neurons.\n",
    "\n",
    "$$Q = \\begin{bmatrix}\n",
    "4 & 1 \\\\\n",
    "8 & 2 \\\\\n",
    "\\end{bmatrix}, $$\n",
    "\n",
    "As you can see if you uncomment and run the next code cell, we get an error if we try to invert this matrix to solve the equation. We'll find out more about this in the next sections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:22.639461Z",
     "iopub.status.busy": "2021-05-30T19:06:22.638012Z",
     "iopub.status.idle": "2021-05-30T19:06:22.639971Z",
     "shell.execute_reply": "2021-05-30T19:06:22.640336Z"
    }
   },
   "outputs": [],
   "source": [
    "# v_p = np.array([16, 7])\n",
    "# Q = np.array([[4, 1], [8, 2]])\n",
    "\n",
    "# print(np.linalg.inv(G) @ L)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1.2: Matrices as linear transformations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:22.644219Z",
     "iopub.status.busy": "2021-05-30T19:06:22.643823Z",
     "iopub.status.idle": "2021-05-30T19:06:22.671569Z",
     "shell.execute_reply": "2021-05-30T19:06:22.672017Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 2: Linear Transformations\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"83AsJJishdM\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Matrices can be thought of as enacting linear transformations. When multiplied with a vector, they transform it into another vector. In fact, they are transforming a grid of space in a linear manner: the origin stays in place and grid lines remain straight, parallel, and evenly spaced.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Coding Exercise 1.2: Creating matrices for transformations\n",
    "\n",
    "Come up with a matrix $A$ for which the corresponding linear transformation is reflection through the $y$ axis (flipping across the $y$ axis). For example, $\\mathbf{x} = \\begin{bmatrix}\n",
    "2 \\\\\n",
    "6  \\\\\n",
    "\\end{bmatrix}$ should become $\\mathbf{b} = \\begin{bmatrix}\n",
    "-2 \\\\\n",
    "6  \\\\\n",
    "\\end{bmatrix}$ when multiplied with $A$. \n",
    "\n",
    "\n",
    "**Remember to think about where your basis vectors should end up! Then your matrix consists of the transformed basis vectors. Drawing out what you want to happen can help**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:22.676119Z",
     "iopub.status.busy": "2021-05-30T19:06:22.674599Z",
     "iopub.status.idle": "2021-05-30T19:06:22.676700Z",
     "shell.execute_reply": "2021-05-30T19:06:22.677155Z"
    }
   },
   "outputs": [],
   "source": [
    "A = ...\n",
    "\n",
    "# Uncomment to visualize transformation\n",
    "#plot_linear_transformation(A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:22.693181Z",
     "iopub.status.busy": "2021-05-30T19:06:22.688384Z",
     "iopub.status.idle": "2021-05-30T19:06:23.207644Z",
     "shell.execute_reply": "2021-05-30T19:06:23.208459Z"
    }
   },
   "source": [
    "[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_a86560cc.py)\n",
    "\n",
    "*Example output:*\n",
    "\n",
    "<img alt='Solution hint' align='left' width=324 height=164 src=https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/tutorials/W0D3_LinearAlgebra/static/W0D3_Tutorial2_Solution_a86560cc_0.png>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Come up with a matrix $A$ for which the corresponding linear transformation is projecting onto the $x$ axis. For example, $\\bar{x} = \\begin{bmatrix}\n",
    "2 \\\\\n",
    "3  \\\\\n",
    "\\end{bmatrix}$ should become $\\bar{b} = \\begin{bmatrix}\n",
    "2 \\\\\n",
    "0  \\\\\n",
    "\\end{bmatrix}$ when multiplied with $A$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:23.211585Z",
     "iopub.status.busy": "2021-05-30T19:06:23.210656Z",
     "iopub.status.idle": "2021-05-30T19:06:23.214492Z",
     "shell.execute_reply": "2021-05-30T19:06:23.215143Z"
    }
   },
   "outputs": [],
   "source": [
    "A = ...\n",
    "\n",
    "# Uncomment to visualize transformation\n",
    "#plot_linear_transformation(A)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:23.218291Z",
     "iopub.status.busy": "2021-05-30T19:06:23.217194Z",
     "iopub.status.idle": "2021-05-30T19:06:23.722063Z",
     "shell.execute_reply": "2021-05-30T19:06:23.722817Z"
    }
   },
   "source": [
    "[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_8d909295.py)\n",
    "\n",
    "*Example output:*\n",
    "\n",
    "<img alt='Solution hint' align='left' width=324 height=164 src=https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/tutorials/W0D3_LinearAlgebra/static/W0D3_Tutorial2_Solution_8d909295_0.png>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Section 1.3: Rank & Null Space"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Video 3: Rank & Null Space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:23.726238Z",
     "iopub.status.busy": "2021-05-30T19:06:23.725183Z",
     "iopub.status.idle": "2021-05-30T19:06:23.756095Z",
     "shell.execute_reply": "2021-05-30T19:06:23.756882Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 3: Rank & Null Space\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"vpl8mz8ji3E\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Square matrices always result in vectors with the same dimensions (number of components) but can alter the dimensionality of the transformed space. Let's say you have a 3 x 3 matrix. You will transform from 3-dimensional vectors to 3-dimensional vectors. You will often be transforming from all of 3D space to all of 3D space. However, this isn't always the case! You could have a 3 x 3 matrix that always results in a vector that lies along a 2D plane through 3D space. This matrix would be transforming from a 3 dimensional vector space (all of R3) to a 2 dimensional vector space (the 2D plane).\n",
    "\n",
    "Matrices that aren't square are enacting transformations that change the dimensionality of the vectors. If you have a 4 x 5 matrix, you are transforming 5-dimensional vectors to 4-dimensional vectors. Similarly, if you have a 4 x 2 matrix, you are transformaing from 4-dimensional vectors to 2-dimensional vectors.\n",
    "\n",
    "The **range of a matrix** is the set of all possible vectors it can lead to after a transformation. The range of the previous example would be the 2D plane. The **rank of a matrix** is the dimensionality of the range: in this case, 2.\n",
    "\n",
    "Sometimes, a matrix will transform a non-zero vector into a zero vector (the origin). The **null space** of a matrix is the set of all vectors that will be transformed into the origin.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Think! 1.3: Neural coding\n",
    "\n",
    "Let's return to the setup of the previous coding exercise: we have two populations of V1 neurons, p and q, responding to LGN neurons. Visualize the linear transformations of these matrices by running the next code cell. Then discuss the following questions:\n",
    "\n",
    "1) What are the ranks of weight matrix P and Q? \n",
    "\n",
    "2) What does the null space of these matrices correspond to in our neuroscience setting? **Advanced:** What do you think the dimensionality of the null space is for P and Q?\n",
    "\n",
    "3) What is the dimensionality of the population of neural responses in population p? How about in q?\n",
    "\n",
    "4) If we wanted to decode LGN neural activity from the V1 activities, would we always be able to completely recover the LGN activity when looking at population p? How about population q? What does this tell us about the information loss of the neural processing?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:23.797549Z",
     "iopub.status.busy": "2021-05-30T19:06:23.787168Z",
     "iopub.status.idle": "2021-05-30T19:06:24.892136Z",
     "shell.execute_reply": "2021-05-30T19:06:24.892602Z"
    }
   },
   "outputs": [],
   "source": [
    "# @markdown Execute to visualize linear transformations\n",
    "P = np.array([[1, 3], [2, 1]])\n",
    "plot_linear_transformation(P, name = 'p')\n",
    "\n",
    "Q = np.array([[4, 1], [8, 2]])\n",
    "plot_linear_transformation(Q, name = 'q')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:24.896042Z",
     "iopub.status.busy": "2021-05-30T19:06:24.895569Z",
     "iopub.status.idle": "2021-05-30T19:06:24.898607Z",
     "shell.execute_reply": "2021-05-30T19:06:24.899041Z"
    }
   },
   "source": [
    "[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_8d0b4dd3.py)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Section 2: Eigenvalues & Eigenvectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:24.903604Z",
     "iopub.status.busy": "2021-05-30T19:06:24.903043Z",
     "iopub.status.idle": "2021-05-30T19:06:24.929725Z",
     "shell.execute_reply": "2021-05-30T19:06:24.930193Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 4: Eigenstuff\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"NLXlYQzQX5Y\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Eigenvectors, $\\mathbf{v}$ of a matrix $\\mathbf{W}$ are vectors that, when multipled by the matrix, equal a scalar multiple of themselves. That scalar multiple is the corresponding eigenvalue, $\\lambda$.\n",
    "\n",
    "$$\\mathbf{W}\\mathbf{v} = \\lambda\\mathbf{v} $$\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Think! 2: Identifying transformations from eigenvectors\n",
    "\n",
    "Earlier, we learned how to think about linear transformations in terms of where the standard basis vectors end up. We can also think about them in terms of eigenvectors. \n",
    "\n",
    "Just by looking at eigenvectors before and after a transformation, can you describe what the transformation is in words? Try for each of the two plots below.\n",
    "\n",
    "Note that I show an eigenvector for every eigenvalue. The x/y limits do not change in before vs after (so eigenvectors are showed scaled by the eigenvalues).\n",
    "\n",
    "Here are some transformation words to jog your memory and guide discussion: contraction, expansion, horizontal vs vertical, projection onto an axis, reflection, and rotation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:24.963063Z",
     "iopub.status.busy": "2021-05-30T19:06:24.947296Z",
     "iopub.status.idle": "2021-05-30T19:06:25.050508Z",
     "shell.execute_reply": "2021-05-30T19:06:25.051281Z"
    }
   },
   "outputs": [],
   "source": [
    "# @title\n",
    "# @markdown Execute this cell to visualize vectors\n",
    "\n",
    "W = np.array([[3, 0], [0, 1]])\n",
    "plot_eig_vec_transform(W)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:25.054498Z",
     "iopub.status.busy": "2021-05-30T19:06:25.053420Z",
     "iopub.status.idle": "2021-05-30T19:06:25.170001Z",
     "shell.execute_reply": "2021-05-30T19:06:25.170707Z"
    }
   },
   "outputs": [],
   "source": [
    "# @title\n",
    "# @markdown Execute this cell to visualize vectors\n",
    "\n",
    "W = np.array([[0, 1], [1, 0]])\n",
    "plot_eig_vec_transform(W)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:25.174047Z",
     "iopub.status.busy": "2021-05-30T19:06:25.172965Z",
     "iopub.status.idle": "2021-05-30T19:06:25.176763Z",
     "shell.execute_reply": "2021-05-30T19:06:25.177392Z"
    }
   },
   "source": [
    "[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_80704181.py)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we saw above, looking at how just the eigenvectors change after a transformation can be very informative about what that transformation was. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Section 3: Matrix multiplication\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "cellView": "form",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:25.180474Z",
     "iopub.status.busy": "2021-05-30T19:06:25.179544Z",
     "iopub.status.idle": "2021-05-30T19:06:25.209286Z",
     "shell.execute_reply": "2021-05-30T19:06:25.209945Z"
    }
   },
   "outputs": [],
   "source": [
    "#@title Video 5: Matrix Multiplication\n",
    "# Insert the ID of the corresponding youtube video\n",
    "from IPython.display import YouTubeVideo\n",
    "video = YouTubeVideo(id=\"x8zuzwwUT2w\", width=854, height=480, fs=1)\n",
    "print(\"Video available at https://youtu.be/\" + video.id)\n",
    "video"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We sometimes want to multiple two matrices together, instead of a matrix with a vector. Let's say we're multiplying matrices $\\mathbf{A}$ and $\\mathbf{B}$ to get $\\mathbf{C}$:\n",
    "$$ \\mathbf{C} = \\mathbf{A}\\mathbf{B}$$.\n",
    "\n",
    "We take the dot product of each row of A with each column of B. The resulting scalar is placed in the element of $\\mathbf{C}$ that is the same row (as the row in A) and column (as the column in B). So the element of $\\mathbf{C}$ at row 4 and column 2 is the dot product of the 4th row of $\\mathbf{A}$ and the 2nd column of $\\mathbf{B}$. We can write this in a formula as:\n",
    "\n",
    "$$\\mathbf{C}_{\\text{row i, column j}} = \\mathbf{A}_{\\text{row i}} \\cdot \\mathbf{B}_{\\text{column j}}$$ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2: Computation corner\n",
    "\n",
    "Break out the pen and paper - it's critical to implement matrix multiplication yourself to fully understand how it works. \n",
    "\n",
    "Let's say we have 3 LGN neurons and 2 V1 neurons. The weight matrix, $W$, between the LGN and V1 neurons is:\n",
    "\n",
    "$$W = \\begin{bmatrix}\n",
    "3 &  2 & 1 \\\\\n",
    "1 & 2 & 7 \\\\\n",
    "\\end{bmatrix}, $$\n",
    "\n",
    "We are going to look at the activity at two time steps (each time step is a column). Our LGN activity matrix, $G$, is: \n",
    "\n",
    "$$ L= \\begin{bmatrix}\n",
    "0 &  1  \\\\\n",
    "2 & 4 \\\\\n",
    "5 & 1 \\\\\n",
    "\\end{bmatrix} $$ \n",
    "\n",
    "Please compute the V1 neural activity, $V$, according to our linear model:\n",
    "\n",
    "$$V = WG $$.\n",
    "\n",
    "\n",
    "Please calculate it 1) by-hand and then 2) using code. Check that the answers match! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:25.212922Z",
     "iopub.status.busy": "2021-05-30T19:06:25.211976Z",
     "iopub.status.idle": "2021-05-30T19:06:25.216180Z",
     "shell.execute_reply": "2021-05-30T19:06:25.216643Z"
    }
   },
   "outputs": [],
   "source": [
    "# Compute by hand first!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:25.219780Z",
     "iopub.status.busy": "2021-05-30T19:06:25.218749Z",
     "iopub.status.idle": "2021-05-30T19:06:25.224376Z",
     "shell.execute_reply": "2021-05-30T19:06:25.225106Z"
    }
   },
   "outputs": [],
   "source": [
    "# Define G\n",
    "G = ...\n",
    "\n",
    "# Define W\n",
    "W = ...\n",
    "\n",
    "# Compute V\n",
    "# in Python, we can use @ for matrix multiplication: matrix1 @ matrix2\n",
    "V = ...\n",
    "\n",
    "# Print values of V\n",
    "print(V)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "execution": {
     "iopub.execute_input": "2021-05-30T19:06:25.227870Z",
     "iopub.status.busy": "2021-05-30T19:06:25.227352Z",
     "iopub.status.idle": "2021-05-30T19:06:25.233225Z",
     "shell.execute_reply": "2021-05-30T19:06:25.234121Z"
    }
   },
   "source": [
    "[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W0D3_LinearAlgebra/solutions/W0D3_Tutorial2_Solution_77c52178.py)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Summary\n",
    "\n",
    "In this tutorial, you have learned how to think about matrices from the perspective of solving a system of equations and as a linear transformation of space. You have learned:\n",
    "- Properties of a matrix, such as rank & null space\n",
    "- How the invertibility of matrices relates to the linear transform they enact\n",
    "- What eigenvalues/eigenvectors are and why they might be useful\n",
    "\n",
    "We will be using this knowledge in many of the days in the NMA computational neuroscience course."
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "include_colab_link": true,
   "name": "W0D3_Tutorial2",
   "provenance": [],
   "toc_visible": true
  },
  "kernel": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
